1. Field of the Invention
The present invention relates to an image synthesis and communication apparatus which continuously captures images by using an imaging apparatus such as a video camera, detects a motion amount of an object image on an imaging plane, synthesizes a wide-angle and high-definition image on the basis of the detected motion amount, and transmits and receives data required for the synthesization.
2. Description of the Related Art
As a prior art, a technique is well known in which, in order to transmit a document image in a TV conference system, an imaging apparatus dedicated to a drawing, using a facsimile apparatus, a scanner, or the like is used in addition to an imaging apparatus for taking an image of attendants.
As another prior art, a technique is well known in which, in order to obtain a wide-angle image in a TV conference system or a TV phone system, the field angle of a lens itself is changed as in the case of a video camera, an imaging apparatus is rotated so as to pan the field of view, or a plurality of still pictures are combined together to be synthesized into one total screen as disclosed in Japanese Unexamined Patent Publication JP-A 59-44184 (1984).
As a further prior art, a technique is well known in which, in order to obtain a high-definition and wide-angle image by an economical imaging apparatus, as disclosed in Japanese Unexamined Patent Publication JP-A 5-260264 (1993), an image of a part of an object is taken with a desired resolution, the object is scanned by rotating or moving the imaging apparatus, and the obtained images are synthesized together.
When images are to be synthesized together, the motion amounts of images must be calculated. This calculation can be performed by several methods.
The representative point method is used mainly in compensation of motion such as a shake of a video camera.
FIG. 19 diagrammatically shows the representative point method. In the method, for continuously captured images, a representative point is set at a fixed position in the image of the previous frame, and, for an image of a current frame, a correlation calculation and an accumulative addition operation are performed on corresponding pixels while conducting a two-dimensional shifting operation. An amount from which the highest calculation value is obtained is detected as a motion amount.
Japanese Unexamined Patent Publication JP-A 6-86149 (1994) discloses a technique in which a Laplacian filter or the like is applied at a preset representative point, a luminance gradient is obtained, and a calculation is performed by using the obtained value, thereby enhancing the accuracy.
A motion amount may be detected also by using the block matching method in which a correlation calculation is performed on duplicated portion of images to determine a position where synthesization is to be conducted. FIG. 20 diagrammatically shows the block matching method. In the method, a specific region which is to be referred is set in the image of the previous frame, a correlation calculation is performed on the image of the current frame while conducting a two-dimensional shifting operation, and a motion amount is obtained in the same manner as the representative point method. In the representative point method, it is required only to obtain correlations from several points and then perform an accumulative addition. By contrast, in the block matching method, an accumulative addition must be performed on all points in the specific region, and hence processing of a higher speed is required.
Among cameras dedicated to a drawing and television cameras used in a facsimile apparatus in the prior art, a low-resolution and narrow-angle camera which is economical has a disadvantage that, in an operation of imaging a document or the like, when an image of the whole of the document is taken, small characters are defaced and illegible, and, when an imaging operation is performed with a resolution at which characters are readable, it is impossible to grasp the whole of the document.
The prior art in which the field angle of a lens itself is changed has a disadvantage that, when the field of view is widened, the resolution is impaired.
The prior art in which an imaging apparatus is rotated has a disadvantage that a wide-angle image cannot be obtained by a single operation.
The technique of Japanese Unexamined Patent Publication JP-A 59-44184 (1984) has a disadvantage that an image cannot be synthesized with a high accuracy and synthesization cannot be attained in an imaging operation in which a hand-held camera is swung. The technique has another disadvantage that a synthesizing apparatus must be disposed in a camera.
In the prior art technique disclosed in Japanese Unexamined Patent Publication JP-A 5-260264 (1993) in which partial images are synthesized together to obtain a high-definition and wide-angle image, there is a disadvantage that, when this technique is combined with a communication system, the transmission side must be provided with a synthesizing apparatus.
Even in the case where synthesization is performed in the receiver side, when all partial images are simply transmitted, data of an amount which is greater than that required in synthesization are transmitted. This produces a disadvantage that the transmission amount is wastefully increased.
Among the methods of detecting a motion amount, in the representative point method, a certain representative point must have a luminance gradient of a given degree. When black and white data such as a document image are to be handled, for example, there arises a problem in that the luminance gradient is low in all representative points and a motion amount cannot be correctly detected.
In other words, the method is effective when the luminance gradient is uniformly distributed in an object image. In the case where the luminance gradient is not uniformly distributed, such as the case of a document having a large background, a low luminance gradient is obtained in the background and hence it is difficult to detect a motion amount.
By contrast, the block matching method has a problem in that the method requires a large amount of calculation and hence it is difficult to detect a motion amount in real time.
It is an object of the invention to provide an image synthesis and communication apparatus which, by using a low-resolution and narrow-angle camera that is economical, can obtain a motion amount between images accurately and rapidly from an object image wherein the luminance gradient is not uniform, such as a document image, synthesize a high-definition and wide-angle image, and handle such an image by means of communication.
In order to attain the object, in a first aspect of the invention there is provided an image synthesis and communication apparatus comprising:
imaging means for inputting images in time series;
image storing means for storing the images inputted from the imaging means;
feature point extracting means for extracting a point of a large luminance change from an image of a current frame, as a feature point;
search range determining means for determining a predetermined region of an image of a subsequent frame, as a search range for a correlation calculation;
correlation calculating means for obtaining correlations between the feature point and pixels in the search range;
image synthesizing means for synthesizing an image on the basis of a motion amount obtained from the correlation calculating means;
image transmitting means for transmitting an image obtained from the image synthesizing means; and
image receiving means for receiving the image transmitted from the image transmitting means.
Preferably, the search range determining means determines the search range on the basis of a motion amount obtained from images of current and previous frames.
Preferably, the feature point extracting means extracts, as a feature point, a point of a large luminance change from plural regions where coordinates in one direction coincide with one another in an image of a current frame, the search range determining means determines a search range for a correlation calculation, for each of the regions of a subsequent frame, and the correlation calculating means performs a correlation calculation for each of the regions.
In another aspect of the invention, in order to attain the object, there is provided an image synthesis and communication apparatus comprising:
imaging means for inputting images in time series;
image transmitting means for transmitting images obtained from the imaging means;
image receiving means for receiving the images transmitted from the image transmitting means;
image storing means for storing the images inputted from the image receiving means;
feature point extracting means for extracting a point of a large luminance change from an image of a current frame, as a feature point;
search range determining means for determining a predetermined region of an image of a subsequent frame, as a search range for a correlation calculation;
correlation calculating means for obtaining correlations between the feature point and pixels in the search range; and
image synthesizing means for receiving a motion amount obtained from the correlation calculating means, and synthesizing an image.
Preferably, the search range determining means determines the search range on the basis of a motion amount obtained from images of current and previous frames.
Preferably, the feature point extracting means extracts, as a feature point, a point of a large luminance change from plural regions where coordinates in one direction coincide with one another in an image of a current frame, the search range determining means determines a search range for a correlation calculation, for each of the regions of a subsequent frame, and the correlation calculating means performs a correlation calculation for each of the regions.
In still another aspect of the invention, in order to attain the object, the image synthesis and communication apparatus comprising:
imaging means for inputting images in time series;
image storing means for storing the images inputted from the imaging means;
feature point extracting means for extracting a point of a large luminance change from an image of a current frame, as a feature point;
search range determining means for determining a predetermined region of an image of a subsequent frame, as a search range for a correlation calculation;
correlation calculating means for obtaining correlations between the feature point and pixels in the search range;
update image calculating means for obtaining an image of a newly-imaged portion from the motion amount obtained from the correlation calculating means;
image transmitting means for transmitting the motion amount obtained from the correlation calculating means, and a partial image obtained from the update image calculating means;
image receiving means for receiving the motion amount and the partial image transmitted from the image transmitting means; and
image synthesizing means for synthesizing an image from the motion amount and the partial image obtained from the image receiving means.
Preferably, the search range determining means determines the search range on the basis of a motion amount obtained from images of current and previous frames.
Preferably, the feature point extracting means extracts, as a feature point, a point of a large luminance change from plural regions where coordinates in one direction coincide with one another in an image of a current frame, the search range determining means determines a search range for a correlation calculation, for each of the regions of a subsequent frame, and the correlation calculating means performs a correlation calculation for each of the regions.
As described above, according to the invention, plural images of a part of an object are taken, the images are synthesized together, and a resulting image is transmitted. Even when a low-resolution and narrow-angle camera is used as the imaging means, therefore, a wide-angle and high-definition image can be transmitted to a remote place.
The place where the image synthesization is performed is not restricted to the transmission side where an image of an object is taken. When obtained images are transmitted as they are from the transmission side and the images are synthesized together in the reception side, a wide-angle and high-definition image can be obtained in the reception side by using a low-resolution and narrow-angle camera and a transmission facility even in the case where the transmission side has only a conventional TV phone system, TV conference system, or the like.
When obtained images are to be transmitted from the transmission side, the obtained images are not transmitted as they are, but only portions of the obtained images which are judged as not overlapping with the previous image by checking the overlapping state of the images with respect to the previous image are transmitted. Therefore, it is possible to transmit only images of a minimum necessary total size, and hence the information amount in the communication can be reduced.
Even in the case where the luminance gradient is not uniformly distributed, a motion amount among images in time series which is basic to the image synthesization can be obtained by a correlation calculation using an arbitrary point as a feature point. Therefore, the motion amount can be accurately detected also from an object having a white background, such as a document. The invention is superior in accuracy than the representative point method.
As compared with the block matching method, the amount of calculation can be reduced by two orders and hence it is possible to perform a real time processing at a frame rate.